{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-02-12T22:17:19.603440Z",
     "start_time": "2019-02-12T22:17:19.523397Z"
    }
   },
   "source": [
    "<h1><center> Facial Emotion Recognition - Boosting on images </center></h1>\n",
    "<center> A project for the French Employment Agency </center>\n",
    "<center> Telecom ParisTech 2018-2019 </center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# I. Context"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The aim of this notebook is to explore facial emotion recognition techniques from a live webcam video stream. \n",
    "\n",
    "The data set used for training is the Kaggle FER2013 emotion recognition data set : https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data\n",
    "\n",
    "The models explored include :\n",
    "- Manual filters \n",
    "- Deep Learning Architectures\n",
    "- DenseNet Inspired Architectures\n",
    "\n",
    "This model will be combined with voice emotion recongition as well as psychological traits extracted from text inputs, and should provide a benchmark and a deep analysis of both verbal and non-verbal insights for candidates seeking for a job and their performance during an interview."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# II. General imports"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Versions used :"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Python : 3.6.5\n",
    "Tensorflow : 1.10.1\n",
    "Keras : 2.2.2\n",
    "Numpy : 1.15.4\n",
    "OpenCV : 4.0.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-12T14:32:42.487934Z",
     "start_time": "2019-03-12T14:32:42.429844Z"
    }
   },
   "outputs": [],
   "source": [
    "### General imports ###\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from time import time\n",
    "from time import sleep\n",
    "import re\n",
    "import os\n",
    "import argparse\n",
    "from collections import OrderedDict\n",
    "import matplotlib.animation as animation\n",
    "\n",
    "### Image processing ###\n",
    "from scipy.ndimage import zoom\n",
    "from scipy.spatial import distance\n",
    "import imutils\n",
    "from scipy import ndimage\n",
    "import cv2\n",
    "import dlib\n",
    "from __future__ import division\n",
    "from imutils import face_utils\n",
    "\n",
    "### CNN models ###\n",
    "import keras\n",
    "from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img\n",
    "from keras.callbacks import TensorBoard\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Dropout, Activation, Flatten\n",
    "from keras.layers.convolutional import Conv2D, MaxPooling2D, SeparableConv2D\n",
    "from keras.utils import np_utils\n",
    "from keras.regularizers import l2#, activity_l2\n",
    "from keras.optimizers import SGD, RMSprop\n",
    "from keras.utils import to_categorical\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "from keras import models\n",
    "from keras.utils.vis_utils import plot_model\n",
    "from keras.layers import Input, GlobalAveragePooling2D\n",
    "from keras.models import Model\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "### Build SVM models ###\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn import svm\n",
    "from lightgbm import LGBMClassifier\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "\n",
    "### Same trained models ###\n",
    "import h5py\n",
    "from keras.models import model_from_json\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# III. Import datas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-12T14:25:36.713555Z",
     "start_time": "2019-03-12T14:25:36.710458Z"
    }
   },
   "outputs": [],
   "source": [
    "path = '/Users/maelfabien/filrouge_pole_emploi/Video/'\n",
    "local_path = '/Users/maelfabien/Desktop/LocalDB/Videos/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-12T14:25:37.256707Z",
     "start_time": "2019-03-12T14:25:36.716880Z"
    }
   },
   "outputs": [],
   "source": [
    "X_train = np.load(local_path + \"X_train.npy\")\n",
    "X_test = np.load(local_path + \"X_test.npy\")\n",
    "y_train = np.load(local_path + \"y_train.npy\")\n",
    "y_test = np.load(local_path + \"y_test.npy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-12T14:25:37.276641Z",
     "start_time": "2019-03-12T14:25:37.263719Z"
    }
   },
   "outputs": [],
   "source": [
    "shape_x = 48\n",
    "shape_y = 48\n",
    "nRows,nCols,nDims = X_train.shape[1:]\n",
    "input_shape = (nRows, nCols, nDims)\n",
    "classes = np.unique(y_train)\n",
    "nClasses = len(classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IV. Build the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-12T14:26:19.187788Z",
     "start_time": "2019-03-12T14:26:19.179848Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(28709, 48, 48, 1)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-12T14:27:54.055640Z",
     "start_time": "2019-03-12T14:27:54.050656Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(28709, 7)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-12T14:59:02.791378Z",
     "start_time": "2019-03-12T14:57:30.796296Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OneVsRestClassifier(estimator=LGBMClassifier(boosting_type='gbdt', class_weight=None, colsample_bytree=1.0,\n",
       "        importance_type='split', learning_rate=0.1, max_depth=-1,\n",
       "        min_child_samples=20, min_child_weight=0.001, min_split_gain=0.0,\n",
       "        n_estimators=10, n_jobs=-1, num_leaves=10, objective=None,\n",
       "        random_state=None, reg_alpha=0.0, reg_lambda=0.0, silent=True,\n",
       "        subsample=1.0, subsample_for_bin=200000, subsample_freq=0,\n",
       "        verbose=1),\n",
       "          n_jobs=None)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = OneVsRestClassifier(LGBMClassifier(learning_rate = 0.1, num_leaves = 10, n_estimators=10, verbose=1))\n",
    "clf.fit(X_train.reshape(-1,48*48*1), y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-12T14:59:02.949914Z",
     "start_time": "2019-03-12T14:59:02.796404Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.008080245193647256\n"
     ]
    }
   ],
   "source": [
    "y_pred = clf.predict(X_test.reshape(-1,48*48*1))\n",
    "print(accuracy_score(y_pred, y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-12T14:56:42.398651Z",
     "start_time": "2019-03-12T14:56:42.386341Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 2, 0, 0, 0, 0, 0])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred>0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# VI. Save the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-07T07:36:40.933059Z",
     "start_time": "2019-03-07T07:36:38.628807Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9521"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#save the model weights\n",
    "json_string = model.to_json()\n",
    "model.save_weights(local_path + 'savedmodels/model_simple_2.h5')\n",
    "open(local_path + 'savedmodels/model_simple_2.json', 'w').write(json_string)\n",
    "#model.save_weights(local_path + 'savedmodels/Emotion_Face_Detection_Model.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-07T07:36:45.396761Z",
     "start_time": "2019-03-07T07:36:42.688716Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded model from disk\n"
     ]
    }
   ],
   "source": [
    "with open(local_path + 'savedmodels/model_simple_2.json','r') as f:\n",
    "    json = f.read()\n",
    "model = model_from_json(json)\n",
    "\n",
    "model.load_weights(local_path + 'savedmodels/model_simple_2.h5')\n",
    "print(\"Loaded model from disk\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# VII. Sources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Visualization : https://github.com/JostineHo/mememoji/blob/master/data_visualization.ipynb\n",
    "- State of the art Architecture : https://github.com/amineHorseman/facial-expression-recognition-using-cnn\n",
    "- Eyes Tracking : https://www.pyimagesearch.com/2017/04/24/eye-blink-detection-opencv-python-dlib/\n",
    "- Face Alignment : https://www.pyimagesearch.com/2017/05/22/face-alignment-with-opencv-and-python/\n",
    "- C.Pramerdorfer,  and  M.Kampel.Facial  Expression  Recognition  using  Con-volutional  Neural  Networks:  State  of  the  Art.  Computer  Vision  Lab,  TU  Wien. https://arxiv.org/pdf/1612.02903.pdf\n",
    "- A Brief Review of Facial Emotion Recognition Based\n",
    "on Visual Information : https://www.mdpi.com/1424-8220/18/2/401/pdf\n",
    "- Going deeper in facial expression recognition using deep neural networks : https://ieeexplore.ieee.org/document/7477450"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
